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The  original  grant  proposal  was  to  advance  Cultural  Consensus  Theory  (CCT)  as  an  information 
pooling  (aggregation,  fusion)  tool,  and  to  examine  possible  areas  where  it  might  be  applicable  to 
military  science.  The  proposal  was  to  conduct  research  in  five  main  project  areas  all  involving  the 
development  of  CCT.  The  main  body  of  this  report  will  describe  these  project  areas  and  the  work 
that  has  been  done  in  each  of  them.  However,  first  some  background  information  about  CCT  will 
be  provided. 

The  PI,  Batchelder,  and  A.  Kimball  Romney  invented  CCT  in  the  1980s  (Batchelder  &  Romney, 
1986,  1988,  1989;  Romney,  Batchelder,  and  Weller,  1987;  Romney,  Weller,  &  Batchelder,  1986), 
and  its  further  development  has  been  supported  between  1986  and  2000  by  several  grants  from  the 
National  Science  Foundation  to  Romney  and  Batchelder,  Co-PIs. 

Data  for  CCT  consists  of  the  responses  of  informants  (experts,  eyewitnesses,  automated  sources,  or 
members  of  a  group)  to  a  set  of  questions  about  some  domain  of  their  shared  knowledge.  Validated 
cognitive  models  are  used  to  estimate  the  consensus  knowledge  of  the  infonnant  group  as  well  as 
the  competence  (degree  of  cultural  knowledge)  and  response  bias  characteristics  of  each  informant. 

As  a  consequence  of  this  earlier  work,  one  of  the  CCT  models  for  dichotomous  True/False  (Yes/No) 
response  data  has  been  in  wide  use  to  detennine  consensus  cultural  views  in  cognitive  and  medical 
anthropology  (see  Weller,  2007).  The  model,  known  as  the  General  Condorcet  Model  (GCM), 
derives  from  assumptions  in  signal  detection  theory  and  psychometric  test  theory,  except  that  unlike 
models  in  these  two  areas,  the  GCM  does  not  assume  that  the  researcher  has  access  to  the  correct 
(consensus)  answers  (signal  or  noise  in  the  case  of  signal  detection,  and  correct  or  error  in  the  case 
of  test  theory).  Instead  the  GCM  postulates  a  consensus  answer  key  Z  =<  Zk  >  as  a  vector  of 
parameters  in  the  model,  where  Zk  =  1  if  the  consensus  answer  to  question  k  is  ‘True’  and  Zk  =  0  if 
‘False’.  In  addition  the  model  postulates  that  each  informant  i  has  a  competence  (ability)  parameter, 
Dj  which  specifies  the  probability  that  he  or  she  detects  the  correct  consensus  answer  to  a  question, 
and  in  addition  each  informant  has  a  guessing  bias  parameter,  gt,  which  governs  the  probability  of  a 
‘True’  response  when  the  correct  answer  is  not  detected.  The  details  of  the  model  are  presented  in 
Batchelder  and  Romney  (1988)  and  Karabatsos  and  Batchelder  (2003),  and  it  is  reviewed  and 
expanded  in  Batchelder  and  Anders  (2012). 

The  main  available  software  package  used  to  estimate  the  GCM  was  developed  in  the  early  1990s 
and  is  found  in  ANTHROPAC  (Borgatti,  1996)  and  UCINET  (Borgatti,  Everett,  &  Freeman,  2002). 
Unfortunately,  this  software  only  provides  point  estimates  of  the  competence  parameters  and  the 
answer  key  parameters  of  a  simplified  version  of  the  GCM  that  sets  the  guessing  bias  to  a  constant 
0.50  for  all  informants.  This  latter  assumption  is  especially  of  concern  because  in  all  applications  of 
signal  detection  models  it  is  well  know  that  different  respondents  have  different  response  bias 
characteristics.  In  Batchelder  and  Romney  (1988),  the  full  GCM  was  developed  that  allowed 
heterogeneity  in  informants,  guessing  bias  as  well  as  heterogeneity  in  item  difficulty  (cultural 
salience);  however,  it  was  much  later  until  Bayesian  inference  software  was  developed  in  S  by 
Karabatsos  and  Batchelder  (2003)  to  estimate  the  full  GCM.  Unfortunately,  Anthropologists 
continue  to  use  the  earlier  software  in  ANTHROPAC  and  UCINET,  and  they  have  not  acquired  the 


1 


programming  skills  or  acceptance  of  Bayesian  inference  theory  needed  to  adopt  that  software. 

The  current  grant  was  primarily  to  develop  new  CCT  models  for  different  questionnaire  formats  as 
well  as  to  establish  state-of-the-art  Bayesian  inference  for  all  the  models  including  the  GCM.  The 
grant  proposal  described  five  main  project  areas  as  follows:  (1)  Constructing  a  catalog  of  CCT 
models  for  different  testing  fonnats;  (2)  Developing  and  implementing  Bayesian  computational 
inference  for  the  models;  (3)  Determining  for  each  testing  format  the  minimal  number  of  informants 
needed  to  achieve  confidence  in  the  pooled  information;  (4)  Developing  an  approach  to  aggregating 
expert  views  of  ties  in  a  digraph  that  imposes  prior  constraints  on  the  consensus  digraph;  (5) 
Developing  CCT  models  that  detect  cultural  variation  and/or  prevarication  among  informants. 

In  the  two  plus  years  of  the  grant,  major  progress  has  been  made  in  project  areas  (1),  (2),  and  (4), 
and  some  progress  has  also  been  made  in  project  areas  (3)  and  (5).  The  next  few  sections  of  this  final 
report  will  describe  the  progress  in  these  five  project  areas  in  detail.  In  addition,  work  supported  by 
this  grant  has  led  to  two  additional  grants:  (1)  A  three  year  grant  to  Batchelder  (PI)  starting  in  the  fall 
of  2010  from  the  Army  Research  Office  (ARO)  to  develop  a  Bayesian  inference  tool  kit  for  CCT 
models  that  is  both  user  friendly  and  freely  available  to  researchers,  and  (2)  A  multi-site  grant  from 
the  Intelligence  Advanced  Research  Projects  Activity  (I  ARP  A)  to  augment  the  probabilistic 
prediction  accuracy  of  government  forecasters.  That  grant  is  funneled  through  Applied  Research 
Associates  (ARA)  and  includes  faculty  researchers  from  several  universities.  Batchelder  is  a 
researcher  on  this  grant,  and  Mark  Steyvers  is  the  UCI  PI. 

Project  Area  1.  Constructing  a  Catalogue  of  CCT  Models, 

As  mentioned  above,  Prior  to  the  grant  from  the  AFOSR  the  main  CCT  model  in  use  was  the  GCM. 
That  model  enabled  researchers  to  ask  a  series  of  True/False  questions  to  the  members  (informants) 
of  a  group  who  share  cultural  knowledge,  and  to  use  the  data  to  infer  the  consensus  answers  to  the 
questions  as  well  as  the  relative  cultural  competence  (level  of  cultural  knowledge)  of  the  informants. 
In  addition,  that  model  was  modified  to  handle  multiple-choice  items,  and  an  informal  CCT 
approach  was  designed  by  Romney,  Batchelder,  and  Weller  (1987)  to  handle  rank  order  response 
data. 

In  the  period  supported  by  the  AFOSR  grant  new  CCT  models  have  been  developed  for  a  number  of 
new  response  modes:  (1)  dichotomous  response  data  based  on  fuzzy  (continuous)  rather  than  crisp 
truth  values  by  Batchelder  and  Anders  (2012)  (2)  reporting  the  ties  in  a  social  network  represented  as 
a  digraph  by  Batchelder,  (2009);  (3)  reporting  the  sign  of  ties  in  a  signed  graph  by  Agrawal  and 
Batchelder  (2012);  (4)  continuous  responses  in  the  unit  interval  (as  in  probability  estimation)  by 
Batchelder,  Strashny,  and  Romney  (2010);  (5)  matching  a  response  set  to  stem  items  (e.g.  flags  to 
countries)  by  Batchelder  and  Steyvers  (in  preparation,  reported  in  several  invited  presentations)  and 
by  Zeigenfuse  and  Batchelder  (in  preparation,  reported  in  several  presentations);  (6)  responses  to 
items  in  an  ordered  categorical  (Likert)  scale  (Anders  and  Batchelder  (2012,  in  preparation);  (7) 
judges  rank  ordering  competitors  by  Anders  and  Batchelder,  (2012,  in  preparation). 

The  ‘in  preparation’  papers  all  have  first  drafts,  but  additional  work  is  needed  before  they  can  be 
submitted  to  a  journal  for  publication.  These  projects  have  gone  a  long  way  to  increasing  the  number 
of  models  for  different  response  types  in  CCT.  Some  of  these  new  models  have  been  published  with 
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new  data  sets,  and  others  are  developed  and  can  handle  recovery  of  parameters  in  simulated  data  but 
we  are  still  seeking  suitable  real  data  to  try  them  out  before  submitting  them  to  a  journal. 

Project  Area  2.  Developing  and  Implementing  Bayesian  Computational  Inference  for  the 
Models. 

Perhaps  the  area  of  the  most  development  during  the  years  of  the  grant  has  been  the  development  of 
Bayesian  inference  software  for  various  CCT  models.  The  basic  work  was  well  underway  during  the 
earlier  years  of  the  grant  from  the  AFOSR,  and  it  picked  up  greatly  with  the  addition  of  the  three 
year  grant  from  the  Army  Research  Office  (ARO)  that  enabled  the  PI  to  hire  a  post  doctoral  fellow 
with  considerable  expertise  in  Bayesian  computational  statistics,  and  to  support  several  graduate 
students  (see  list  of  laboratory  personnel  dealing  with  CCT  in  a  later  section). 

As  mentioned,  Bayesian  inference  for  the  GCM  is  in  Karabatsos  and  Batchelder  (2003).  That  paper 
developed  a  Markov  Chain  Monte  Carlo  (MCMC)  sampler  for  the  GCM  in  the  S  language  (a 
forerunner  of  R).  It  was  a  hybrid  Metropolis-Hastings  and  Gibbs  sampler,  and  code,  but  no  interface, 
was  offered  to  potential  users.  Unfortunately  only  a  few  researchers  have  used  the  software,  and 
instead  they  have  continued  to  use  CCT  software  developed  in  the  late  1980s  and  provided  in 
ANTHROPAC  (Borgatti,  1996)  and  UCINET  (Borgatti,  Everett,  &  Freeman,  2002).  That  software 
was  developed  using  results  in  Batchelder  and  Romney  (1988).  The  key  to  the  software  is  to  obtain  a 
point  estimate  of  each  informant’s  competency  using  a  methods-of-moments  approach,  and  then  use 
these  point  estimates  along  with  Bayes  theorem  to  obtain  a  posteriori  distribution  over  the  possible 
answer  keys.  There  are  two  salient  disadvantages  in  using  that  software.  First,  as  mentioned,  it  only 
applies  to  a  special  case  of  the  GCM  for  T/F  items,  where  there  is  no  informant  heterogeneity  in 
guessing  bias  and  there  is  no  heterogeneity  in  item  difficulty.  Second,  with  only  point  estimates  of 
the  competencies  of  the  informants,  there  is  no  way  to  assess  the  confidence  in  these  point  estimates. 
The  Karabatsos  and  Batchelder  (2003)  solved  these  problems;  however,  as  stated  earlier,  the  social 
science  community  that  uses  CCT  has  not  adopted  that  approach.  One  of  the  main  thrusts  of  the 
grant  was  to  correct  this  situation. 

All  of  the  CCT  papers  during  the  duration  of  the  grant  have  used  Bayesian  inference  via  MCMC 
samplers.  Some  of  the  models  were  sufficiently  complex  that  we  had  to  develop  our  own  MCMC 
samplers,  e.g.  Batchelder,  Strashny,  and  Romney  (2010),  and  Agrawal  and  Batchelder  (2012). 
However,  the  major  breakthrough  during  the  grant  was  to  show  that  standard,  free  access  software 
packages  could  handle  Bayesian  inference  for  the  most  general  version  of  the  GCM.  In  particular, 
we  developed  software  with  JAGS  that  uses  R  in  Batchelder  and  Anders  (2012),  and  more  recently 
we  have  designed  a  software  package  for  the  GCM  that  uses  WinBUGS  (Lund,  Thomas,  Best,  & 
Spiegelhalter,  2000).  That  package  is  described  in  Oracevz,  Vandekerchove,  and  Batchelder  (2012). 
Both  papers  are  under  review.  The  latter  paper  designs  the  package  BCCT  (Bayesian  Cultural 
Consensus  Toolbox)  that  can  analyze  eight  different  versions  of  the  GCM  depending  on  whether  or 
not  competence,  guessing  bias,  and  item  difficulty  are  heterogeneous.  It  is  designed  to  be  very  user 
friendly,  and  the  paper  is  designed  to  allow  a  reader  to  download  and  use  the  package. 

Project  Area  3.  Determining  for  each  Testing  Format  the  Minimal  Number  of  Informants 
Needed  to  Achieve  Confidence  in  the  Pooled  Information 


3 


In  Batchelder  and  Romney  (1988),  power  tables  were  provided  for  the  GCM  (the  CCT  model  for 
T/F  items)  that  gave  the  minimum  number  of  informants  needed  to  achieve  a  satisfactory  recovery  of 
the  consensus  answers  to  the  T/F  items  as  a  function  of  the  average  competence  and  pre-specified 
levels  of  confidence.  One  surprising  aspect  of  this  table  is  that  far  fewer  informants  are  needed  to 
achieve  a  particular  level  of  confidence  in  the  recovery  of  the  consensus  answers  based  on  a  majority 
rule  with  confidence  intervals  from  the  binomial  theorem.  This  is  because  the  GCM  endogenously 
weights  the  informants  by  their  competence,  and  gives  more  weight  to  the  more  competent 
infonnants.  As  shown  in  Batchelder  and  Romney  (1988)  this  leads  to  log-odds  aggregation  rather 
than  simple  majority  rule. 

The  goal  of  this  project  area  is  to  develop  similar  tables  for  other  CCT  models.  As  mentioned  in 
Project  Area  #1,  several  new  CCT  models  have  been  developed  during  the  grant.  We  have 
conducted  numerous  simulations  using  Bayesian  software  on  simulated  data  for  these  models.  These 
simulations  have  varied  the  number  of  informants  and  their  average  competence  toward  the  aims  of 
this  Project  Area.  While  we  are  not  yet  able  to  present  detailed  tables  for  the  new  models  in  the  form 
of  the  table  for  the  GCM  in  Batchelder  and  Romney  (1988),  it  is  clear  from  this  work  that  for 
response  models  more  complex  than  dichotomous  T/F  items,  even  fewer  informants  are  necessary  to 
achieve  accurate  reconstruction  of  the  consensus  answers. 

In  fact,  if  the  assumption  of  a  single  consensus  truth  is  valid  for  a  given  set  of  response  data,  once 
the  number  of  informants  exceeds  ten  or  so,  one  can  operate  with  a  CCT  model  at  about  the  same 
level  of  precision  as  would  be  had  in  case  one  had  the  consensus  answers  apriori.  For  this  reason  it 
has  become  of  paramount  impotents  to  develop  tests  of  the  single  culture  assumption.  Such  a  test 
was  developed  as  a  Bayesian  post  predictive  test  in  Batchelder  and  Anders  (2012).  The  test  is  based 
on  a  property  of  the  GCM  proved  in  Batchelder  and  Anders  (2012)  that  the  correlation  between  two 
infonnants,  i  and  j,  over  their  responses  to  the  items,  ptj,  satisfies  the  formula  />,,  =  pi7pj7 ,  where 

the  right  hand  side  is  the  product  of  the  conelation  of  each  informant  with  the  unknown  consensus 
answers.  Such  a  fact  is  a  version  of  Spearman’s  famous  tetrad  law,  pijpu  =  pnpkj ,  concerning 

correlations  between  tests  across  test  takers  that  is  behind  his  two  factor  theory  of  intelligence.  In 
essence  this  property  for  CCT  models  says  that  the  sole  basis  of  any  correlation  between  informants 
is  due  entirely  to  the  fact  that  they  are  hypothesized  to  share  a  common  consensus  answer  key.  This 
result  suggests  that  if  the  informant-by-informant  correlations  are  subjected  to  a  factor  analysis,  that 
the  signal  in  the  data  will  be  represented  in  the  first  factor  as  shown  above.  The  Bayesian  post 
predictive  check  of  the  GCM  model  compares  the  ratio  of  the  first  to  second  eigenvalues  of  the 
factor  analysis  for  the  real  data  against  the  same  ratio  obtained  by  simulating  thousands  of  simulated 
data  sets  obtained  during  the  MCMC  sampler.  The  property  of  a  single  consensus  answer  key  stands 
or  falls  with  rather  or  not  the  ratio  in  the  real  data  falls  midway  in  the  distribution  of  ratios  from  the 
simulated  data  sets. 

Project  Area  4.  Developing  an  approach  to  aggregating  expert  views  of  ties  in  a  digraph  that 
imposes  prior  constraints  on  the  consensus  digraph. 

Batchelder,  Kumbasar,  and  Boyd  (1997)  and  Batchelder  (2009)  provide  CCT  models  for  pooling 
responses  from  experts  about  the  arcs  in  a  digraph.  In  these  models,  there  are  no  prior  constraints  on 
the  nature  of  the  responses  of  the  experts  such  as  symmetry,  transitivity,  or  structural  balance. 
Imposing  constraints  on  the  consensus  graph,  without  at  the  same  time  imposing  them  on  the 
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experts’  responses  leads  to  the  necessity  of  developing  a  special  MCMC  sampler  to  do  Bayesian 
inference.  This  is  because  available  software  packages  like  JAGS  and  WinBUGS  cannot  impose 
structural  constraints  in  graphs  in  their  samplers. 

In  the  period  under  review  we  have  worked  with  signed  graphs  (graphs  that  have  symmetric  ties  that 
are  either  positive  or  negative,  such  as  friend  or  enemy).  Agrawal  and  Batchelder  (2012)  present  a 
model  for  the  case  where  each  expert  provides  a  complete  signed  graph  (there  is  a  signed  tie  between 
every  pair  of  distinct  nodes),  and  the  consensus  graph  is  required  to  satisfy  Cartwright  and  Harary’s 
definition  of  balance  in  a  social  network.  This  sense  of  balance  is  that  the  graph  nodes  can  be 
partitioned  into  two  sets,  one  of  which  may  be  empty,  where  ties  between  nodes  in  the  same  set  are 
positive  and  ties  between  nodes  in  different  sets  are  negative.  If  the  graph  has  N  nodes,  it  is  easy  to 
show  that  there  are  2A(A~11,2  distinct  complete  signed  graphs,  and  only  2  V  1  of  them  satisfy  this  sense 
of  balance.  The  model  new  model  was  used  to  analyze  real  and  simulated  data,  and  the  MCMC 
sampler  that  was  developed  is  able  to  recover  the  expert  and  item  parameters  as  well  as  the 
consensus  balanced  graph.  This  work  has  been  presented  in  three  conference  papers,  two  at  the  2012 
Annual  Sunbelt  Social  Network  conference  and  one  at  the  2012  Annual  Social  Computing, 
Behavioral  Cultural  Modeling,  and  Prediction  conference. 

The  special  concern  in  imposing  constraints  on  a  consensus  graph  structure  is  that  there  must  be  a 
way  to  search  the  parameter  space  of  graphic  structures  that  satisfy  the  constraint.  This  can  lead  to 
complex  combinatorial  issues.  For  example,  the  work  in  Agrawal  and  Batchelder  (2012)  is  being 
generalized  to  Davis’  sense  of  balance,  where  the  partition  can  have  more  than  two  cells,  where  as 
before  positive  ties  are  within  cells  and  negative  ties  between  cells.  In  this  case  the  so-called  Bell 
number  gives  the  number  of  partitions  on  a  finite  set,  and  this  number  can  only  be  expressed 
recursively.  Similar  combinatorial  complexity  issues  occur  with  other  constraints  such  as  transitivity 
or  a  partial  order. 

It  is  supposed  that  CCT  models  for  pooling  graphs  under  constraints  may  be  useful  in  detecting 
intelligence  applications,  for  example  detecting  covert  networks.  In  such  a  case,  the  intelligence 
experts  will  use  their  knowledge  to  fill  in  the  arcs  in  a  covert  network  defined  by  ‘giving  orders  to,’ 
‘meeting  secretly  with’,  or  ‘sharing  information  with.’  Then  one  would  impose  certain 
organizational  structures  on  the  consensus  graph  based  on  intelligence  reports  about  the  likely 
structure  of  a  particular  covert  network,  for  example  a  central  node  representing  the  coordinator  with 
arcs  to  several  others  to  the  members,  but  no  other  arcs  between  the  members  to  preserve  secrecy. 
The  value  of  the  approach  is  that  even  though  the  experts’  graphic  responses  do  not  satisfy  the 
supposed  constraints,  perhaps  due  to  incomplete  knowledge,  the  resulting  consensus  graph  will 
represent  the  most  likely  covert  structures.  It  is  planned  to  seek  further  funding  to  carry  out  this 
program.  It  will  involve  complex  issues  in  combinatorial  complexity  and  special  MCMC  samplers. 

Project  Area  5.  Developing  CCT  models  that  detect  cultural  variation  and/or  prevarication 
among  informants. 

Most  of  the  CCT  models  are  designed  for  the  situation  where  all  the  informants  share  the  same 
consensus  answers  to  the  questions.  This  crucial  one-culture  property  of  the  GCM  for  dichotomous 
T/F  questions  is  essential  for  the  Bayesian  inference  for  the  model  provided  in  Karabatsos  and 
Batchelder  (2003),  and  it  lead  to  an  important  Bayesian  post  predictive  check  developed  in 
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Batchelder  and  Anders  (2012)  discussed  earlier  in  project  area  3. 

The  most  natural  way  to  relax  the  one-culture  assumption  and  allow  for  cultural  variation  is  to 
augment  the  model  to  allow  two  or  more  consensus  answer  keys.  The  formalization  of  this  idea  as  a 
finite  mixture  model  was  presented  in  Batchelder  and  Romney  (1989);  however,  no  complete 
statistical  inference  was  provided.  Their  approach  to  augmenting  the  GCM  was  to  add  two  or  more 
answer  keys  along  with  informant  membership  parameters  that  indicate  which  answer  key  belongs  to 
which  informant. 

In  the  period  under  review,  the  GCM  model  developed  earlier  has  been  reworked  so  that  Bayesian 
inference  for  the  model  allowing  up  to  three  answer  keys  can  be  conducted  with  JAGs,  the  same 
Bayesian  software  package  used  in  Batchelder  and  Anders  (2012)  to  estimate  the  GCM  with  a  single 
consensus  answer  key.  So  far  a  draft  of  a  paper  has  been  written  and  a  data  set  involving  several 
answer  keys  is  being  analyzed.  We  expect  that  this  paper  will  be  completed  and  submitted  for 
publication  by  June  2012. 

Crucial  in  this  work  is  to  develop  a  diagnostic  for  the  number  of  answer  keys  that  are  needed  to  best 
fit  the  data.  Toward  this  goal,  a  new  theorem  for  the  multi-culture  GCM  has  been  derived.  The 
theorem  concerns  the  correlations  between  pairs  of  informants  over  the  items,  /^introduced  earlier 

in  progress  area  3.  The  result  is  for  all  pairs  of  informants  i  and  j,  p(j  -  PiZu)Pza)zu)PjZu)-  di  this 

formula  the  first  and  last  term  refer  to  the  correlation  between  one  of  the  informants’  responses  to 
the  items  and  his  or  her  own  consensus  answer  key,  and  the  middle  term  is  the  correlation  between 
the  two  answer  keys.  In  fact  this  formula  reduces  to  the  one  described  in  project  area  3  if  the  two 
infonnants  share  the  same  consensus  answer  key.  We  have  constructed  a  test  of  multi-cultures  based 
on  this  formula.  The  idea  is  that  when  the  informant-by-informant  correlation  matrix  is  factored  (see 
project  area  3),  the  first  factor  will  underestimate  correlations  between  infonnants  with  the  same 
consensus  answer  key  and  overestimate  the  correlation  between  informants  with  different  answer 
keys.  Based  on  this,  it  is  possible  to  cluster  the  informants  based  on  the  second  residual  matrix  after 
the  first  factor  is  pulled  out  of  the  original  correlation  matrix.  We  are  working  this  idea  into  a  new 
Bayesian  post  predictive  check  for  the  multi-culture  version  of  the  GCM. 

While  the  expansion  of  the  model  to  handle  multiple  answer  keys  enables  us  to  detect  cultural 
variation  when  there  is  more  than  one  informant  in  a  cultural  group,  we  have  made  less  progress  in 
detecting  prevarication  in  infonnants  as  described  in  the  second  part  of  the  goals  of  project  area  5. 
Currently  we  are  working  with  a  psychometrician  to  adapt  an  approach  used  in  item  response  theory 
(IRT)  called  differential  item  functioning  (DIF)  that  is  designed  to  find  respondents  who  have 
different  response  probabilities  than  the  large  majority  of  the  other  respondents.  Our  goal  is  to 
discover  a  Tie  detection  statistic’  that  can  be  used  to  screen  the  informant  pool.  We  are  still  working 
on  this  problem  at  this  time. 

In  short,  there  has  been  a  great  deal  of  progress  in  the  five  project  areas  of  the  grant,  and  this  has  led 
to  a  number  of  papers,  invited  conference  and  workshop  presentations,  several  additions  to  the  Pis 
lab,  and  two  new  grants  concerning  CCT. 


Personal  During  the  AFOSR  Grant 
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Because  of  the  support  of  the  grant  from  the  AFOSR,  I  have  added  several  graduate  students  and 
post  doctoral  researchers  to  my  laboratory  who  I  advise  and  are  working  on  CCT  projects.  I  list  them 
and  any  change  in  their  status  during  the  period  of  the  grant. 


1 .  Royce  Anders-  Passed  his  PhD  Candidacy  Exam  in  Department  of  Cognitive  Sciences,  currently 
a  fourth  year  student. 

2.  Gregory  Alexander-  Started  as  an  Undergraduate  Researcher  on  the  grant  and  is  now  a  first  year 
graduate  student  in  the  PhD  Program  of  the  Department  of  Cognitive  Sciences. 

3.  Kalin  Agrawal-  Third  year  graduate  student  in  the  Mathematical  Behavioral  Sciences  PhD 
Program. 

4.  Giorgio  Gosti-  Passed  his  PhD  Candidacy  Exam  in  the  Mathematical  Behavioral  Sciences  PhD 
Program. 

5.  Dr.  Zita  Oravecz,  Post  Doctoral  Fellow  who’s  PhD  was  from  University  of  Leuven,  Belgium, 
supported  by  the  grant  from  Army  Research  Office  described  earlier. 

6.  Dr.  Stephen  France,  Assistant  Professor  Market  Research  University  of  Wisconsin  Milwaukee. 
Sabbatical  to  learn  about  CCT  and  apply  it  to  market  researach 

Publications  During  the  AFOSR  Grant  (*  indicates  ones  directly  concerned  with  CCT) 


*Batchelder,  W.  H.  (2009).  Cognitive  Pscyometrics:  Using  Multinomial  Processing  Tree  Models 
as  Measurement  Tools.  In  S.  E.  Embretson,  Measuring  Psychological  Constructs:  Advances  in 
Model  Based  Measurement  (pp.  71-93).  Washington  D.C.:  American  Psychological  Association 
Books. 

*Batchelder,  W.  H.  (2009).  Cultural  Consensus  Theory:  Aggregating  Expert  Judgments  about  Ties 
in  a  Social  Network.  In  H.  Liu,  J.  Salerno,  &  M.  J.  Young,  Social  Computing,  Behavioral 
Modeling  and  Prediction  (pp.  24-32).  New  York:  Springer. 

Batchelder,  W.  H.,  Hu,  X.,  &  Smith,  J.  B.  (2009).  Multinomial  Processing  Tree  Models  for 
Discrete  Choice.  Special  Issue  on  New  Developments  in  Multinomial  Process  Tree  Modeling. 
Zeitschrift  fur  Psychologie  ,  217,  149-158. 

Purdy,  B.,  &  Batchelder,  W.  H.  (2009).  A  Context-free  Language  for  Binary  Multinomial 
Processing  Tree  Models.  Journal  of  Mathematical  Psychology ,  53,  547-561. 

Batchelder,  W.  H.  (2010).  Mathematical  Psychology.  In  L.  Nadel,  Wiley  Interdisciplinary 
Reviews:  Cognitive  Science  (pp.  759-765).  New  York:  Wiley. 

*Batchelder,  W.H.,  Strashny,  A.,  and  Romney,  A.K.  (2010)  Cultural  Consensus  Theory: 
Aggregating  Continuous  Responses  in  a  Finite  Interval.  In  S.-K.  Chai,  J.J.  Salerno,  and  P.L.  Mabry 
(Eds.).  Social  Computing,  Behavioral  Modeling,  and  Prediction  2010  (pp.  98-107).New  York: 
Springer,  2010,  pp.  98-107. 

Smith,  J.B.  and  Batchelder,  W.H.  (2010).  Beta-MPT:  Multinomial  processing  tree  models  for 
addressing  individual  differences.  Journal  of  Mathematical  Psychology,  54,  167-183. 
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Wu,  H.,  Myung,  J.I.,  and  Batchelder,  W.H.  (2010).  On  the  minimum  description  length  complexity 
of  multinomial  processing  tree  models.  Journal  of  Mathematical  Psychology,  54,  291-303. 

Wu,  H.,  Myung,  J.I.,  and  Batchelder,  W.H.  (2010).  Minimum  description  length  model  selection  of 
multinomial  processing  tree  models.  Psychonomic  Bulletin  &  Review,  17,  275-286. 

Schmittmann,  V.D.,  Dolan,  C.V.,  Raijmakers,  M.E.J.,  and  Batchelder,  W.H.  (2010).  Parameter 
identification  in  multinomial  processing  tree  models.  Behavior  Research  Methods \  42,  836-846. 

Batchelder,  W.H.  (2010).  Mathematical  Psychology.  In  L.  Nadel  (Ed.) .Wiley  Interdisciplinary 
Reviews:  Cognitive  Sciences  (pp.  759-765).  New  York:  Wiley. 

*Gosti,  G.,  and  Batchelder,  W.H.  (2011).  Naming  on  a  directed  graph.  In  J.J.  Salerno,  J.Y. 
Shanchieh,  D.S.  Nau,  and  S-K  Chai  (Eds.).  Social  Computing,  Behavioral-Cultural  Modeling  and 
Prediction  LNCS  6589  (pp.  358-365).  New  York:  Springer  Verlag. 

*Agrawal,  K.,  and  Batchelder,  W.H.  (2012).  Cultural  consensus  theory:  Aggregating  signed  graphs 
under  a  balance  constraint.  In  S.J.  Yang,  A.  M.  Greenberg,  and  M.  Endsley  (Eds.).  Social 
Computing,  Behavioral-Cultural  Modeling  and  Prediction,  LNCS  7227  (pp. 53-60).  New  York: 
Springer  Verlag. 

Batchelder,  W.H.,  Hu,  X.,  and  Riefer,  D.M.  Multinomial  Modeling.  In  H.  Pashler  (Ed.).  The 
Encyclopedia  of  the  Mind.  Sage  Publications,  in  press. 

Invited  Conference  and  Workshop  Presentations  on  CCT  During  the  AFOSR  Grant 

Batchelder,  W.H.  Invited  paper  on  Cultural  Consensus  Theory  read  at  AFOSR  Program  Review  of 
Mathematical  Modeling  of  Cognitive  and  Decision  Processes.  Arlington  Vg.,  January  2009. 

Batchelder,  W.H.  Cultural  Consensus  Theory:  Aggregating  expert  judgments  about  ties  in  a  Social 
Network.  Invited  paper  Read  at  2nd  Annual  Workshop  on  Social  Computing,  Behavioral 
Modeling,  and  Prediction.  Phoenix,  Az.  April  2009. 

Batchelder,  W.H.  Cultural  Consensus  Theory:  New  models  for  continuous  response  and  matching 
tests.  AFOSR  Joint  Review  Cognition  and  Decision  Making  Program  and  Human-System 
Interface  Program.  Arlington,  Vg.,  January  2010. 

Batchelder,  W.H.,  Strashny,  A.,  and  Romney,  A.K.  Cultural  Consensus  Theory:  A  model  for  a 
continuous  responses  in  a  finite  interval.  Invited  Paper  read  at  the  Annual  Conference  on  Social 
Computing,  Behavioral  Modeling,  and  Prediction.  NIH  Campus,  Bethesda,  Md.  April  2010. 

Batchelder,  W.H.  Cultural  Consensus  Theory.  Invited  paper  read  to  RAND  Corporation  Centers, 
Santa  Monica,  Ca.  July  2010. 

Batchelder,  W.H.  Cultural  Consensus  Theory.  Invited  paper  read  in  Symposium  on  Wisdom  of  the 
Crowd,  Annual  Meeting  of  the  Society  for  Mathematical  Psychology,  Portland,  Or.  August  2010. 
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Anders,  R.  (Presenter),  and  Batchelder,  W.H.  Rank-Aggregation  and  consensus  in  Ballroom 
Competition.  Paper  read  at  Annual  Meeting  of  the  Society  for  Mathematical  Psychology,  Portland, 
Or.  August  2010. 

Zeigenfuse,  M.  (Presenter),  Batchelder,  W.H.,  and  Steyvers,  M.  A  three  parameter  Item  Response 
Model  of  Matching.  Paper  read  at  Annual  Meeting  of  the  Society  for  Mathematical  Psychology, 
Portland,  Or.  August  2010. 

Batchelder,  W.H.  Statistical  Development  and  Applications  of  Cultural  Consensus  Theory.  Paper 
presented  at  AFOSR  Joint  Review  Cognition  and  Decision  Making  Program.  Dayton,  Ohio, 
January  2011. 

Batchelder,  W.H.  Observations  about  Cultural  Consensus  Theory.  Paper  presented  at 
Workshop  on  Dynamic  Models  of  Cultural  Diversity.  Arizona  State  University,  Tempe, 

Az.  February,  2011 

Gosti,  G.  (presenter),  and  Batchelder,  W.H.  Naming  on  a  directed  graph.  Poster  presented  at  the 
Conference  on  Social  Computing,  Behavioral-Cultural  Modeling,  and  Prediction.  University  of 
Maryland,  College  Park,  Md.  March  2011. 

Batchelder,  W.  H.,  and  Anders,  R.  Cultural  Consensus  Theory:  Comparing  different  concepts  of 
cultural  truth.  Paper  presented  at  Annual  Meeting  of  the  Society  for  Mathematical  Psychology. 
Tufts  University,  July  2011. 

Gosti,  G.  (presenter),  and  Batchelder,  W.H.  The  Naming  Game  on  a  Directed  Graph.  Paper 
presented  at  Annual  Meeting  of  the  Society  for  Mathematical  Psychology.  Tufts  University,  July 
2011. 

Agrawal,  K.  (presenter),  and  Batchelder,  W.H.  Cultural  Consensus  Theory:  Estimating  Consensus 
Graphs  under  constraints.  Paper  presented  at  Annual  Meeting  of  the  Society  for  Mathematical 
Psychology.  Tufts  University,  July  2011. 

Batchelder,  W.H.  Cultural  Consensus  Theory:  Detecting  Experts  and  Their  Shared  Knowledge. 
Invited  Paper  presented  at  DIMACS  Workshop  on  the  Science  of  Expert  Opinion.  Rutgers 
University,  October,  2011. 

Papers  Under  Review  on  CCT 

Batchelder,  W.H.,  and  Anders,  R.  (2012).  Cultural  consensus  theory:  Comparing  different 
concepts  of  cultural  truth.  Journal  of  Mathematical  Psychology  _( under  review). 

Oravecz,  Z.,  Vandekerckhove,  and  Batchelder,  W.  H.  (2012).  Bayesian  cultural  consensus  theory. 
Field  Methods  (under  review) 

Additional  Research  Opportunities  Made  Possible  by  the  Grant. 
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There  were  several  research  connections  between  our  lab  and  others  that  were  facilitated  by  the 
grant.  They  are  covered  in  the  next  few  paragraphs. 

Researchers  at  RAND  in  Santa  Monica  expressed  interest  in  the  CCT  project.  They  invited  the  PI, 
Batchelder,  to  give  an  hour  talk  on  CCT  that  was  broadcasted  to  all  the  divisions  of  RAND, 
Batchelder,  W.H.  Cultural  Consensus  Theory.  Invited  paper  read  to  RAND  Corporation  Centers, 
Santa  Monica,  Ca.  July  2010.  In  addition,  RAND  hired  Kalin  Agrawal,  an  advanced  PhD  student 
in  the  Pis  lab  for  a  summer  internship  at  RAND. 

The  Army  Research  Office  representatives  heard  a  talk  by  the  PI  on  CCT,  and  after  some 
correspondence  they  funded  a  three-year  grant  for  over  $300,000  that  is  in  its  second  year.  This 
grant  allowed  the  hiring  of  a  Post  Doctoral  fellow,  Zita  Oravecz,  who  has  considerable  skill  with 
software  and  Bayesian  statistics.  This  has  led  to  the  software  package,  BCCT,  described  in 
Oravecz,  Vandekerckhove,  and  Batchelder  (2012). 

An  Institute  for  Cultural  Studies  at  Arizona  State  University  funded  a  Workshop  on  Dynamic 
Models  of  Cultural  Diversity.  The  PI  was  a  keynote  presenter  to  acknowledge  the  25th  anniversary 
of  the  development  of  CCT.  The  efforts  of  the  researchers  at  this  workshop  have  led  to  funding 
opportunities  for  the  group  with  NSF.  The  invited  presentation  was:  Batchelder,  W.H. 
Observations  about  Cultural  Consensus  Theory.  Paper  presented  at  Workshop  on  Dynamic  Models 
of  Cultural  Diversity.  Arizona  State  University,  Tempe,  Az.  February,  2011. 

A  multi-site  grant  for  several  million  dollars  was  received  from  the  Intelligence  Advanced  Research 
Projects  Activity  (IARPA)  to  augment  the  probabilistic  prediction  accuracy  of  government 
forecasters.  That  grant  is  funneled  through  Applied  Research  Associates  (ARA)  and  includes  faculty 
researchers  from  several  universities.  Batchelder  is  a  researcher  on  this  grant,  and  Mark  Steyvers  is 
the  UCI  PI. 
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